Falling and heaviness: Heaviness judgment for a visual object which users lift up is influenced by the presentation of the object's falling or staying still.

When lifting and subsequently releasing a visual object on a screen using a computer mouse, users tend to judge the object to be heavier when the motion speed of the object during lifting is smaller. However it was unclear how the presentation of an object falling after its release influenced the judgment of heaviness. Users generally believe mistakenly that heavier objects fall faster. Based on the previous report of this misbelief, we briefly explored how the falling speed of a visual object after release by a user influenced the judgment of heaviness. The falling speed of the object was systematically modulated by changing gravity in the simulation of the natural falling of the object. Participants judged the object's heaviness after they lifted and subsequently released it. As a result, the participants judged the object to be lighter when the falling speed was zero. However, no significant differences were observed among the conditions with a falling speed greater than zero. It is suggested that for the judgment of heaviness, a vital aspect in the presentation of a falling object after releasing is whether the object falls or not.

Specifying Visual Parameters for Haptic-Visual Sequential Matching of Material Softness.

When shopping online, a customer cannot directly touch the products but may sometimes make judgments about the haptic properties of a product based only on visual information, before making a purchase decision. In this scenario, a customer may be dissatisfied if there is an inconsistency in the judgment of the product's haptic properties they made before purchasing, and their actual experience of those haptic properties once they have received the product. Thus, it is necessary for online sellers to appropriately optimize visual information for materials so that perceived softness is consistent between haptic and visual modalities presented in different locations and at different moments in time. Focusing on visual indentation depth and speed, we examined the visual parameters used to sequentially match haptic and visual softness from haptic and visual information made available in different locations and at different times. Participants performed a two-alternative forced choice task to determine which of two video clips contained an elastic material with a softness impression most similar to the haptic softness of an actual material that the participants indented with their index finger. Based on a sequence of 25 repeated judgments for each material, our algorithm optimized each visual parameter based on a Gaussian process. The optimized visual indentation depth varied consistently with material compliance, while the optimized visual indentation speed did not, suggesting that visual indentation depth was critical for softness matching. The optimized visual indentation depth was highly correlated with the haptic indentation depth. Subjective rating scores for the softness matching increased significantly after the optimization process. The results indicate that participants are able to successfully match the haptic and visual softness of materials by checking the relationship between indentation depths detected haptically, and those detected visually.

Softness Perception of Visual Objects Controlled by Touchless Inputs: The Role of Effective Distance of Hand Movements.

Feedback on the material properties of a visual object is essential in enhancing the users' perceptual experience of the object when users control the object with touchless inputs. Focusing on the softness perception of the object, we examined how the effective distance of hand movements influenced the degree of the object's softness perceived by users. In the experiments, participants moved their right hand in front of a camera which tracked their hand position. A textured 2D or 3D object on display deformed depending on the participant's hand position. In addition to establishing a ratio of deformation magnitude to the distance of hand movements, we altered the effective distance of hand movement, within which the hand movement could deform the object. Participants rated the strength of perceived softness (Experiments 1 and 2) and other perceptual impressions (Experiment 3). A longer effective distance produced a softer impression of the 2D and 3D objects. The saturation speed of object deformation due to the effective distance was not a critical determinant. The effective distance also modulated other perceptual impressions than softness. The role of the effective distance of hand movements on perceptual impressions of objects under touchless control is discussed.

Pseudo-Haptic Heaviness Influenced by the Range of the C/D Ratio and the Position of the C/D Ratio Within a Given Range.

Pseudo-haptic heaviness refers to the illusory sensation of heaviness caused by a dissociation in amplitudes between object movements on a screen and users' motor actions. The amplitude ratio of object movements to the user's actions, the so-called C/D ratio, is a powerful determinant of pseudo-haptic heaviness. According to previous studies, perceptual judgments for a given stimulus value are influenced by the position of the value within a given stimulus range, while no studies have shown the same to be true for pseudo-haptic heaviness. The present study examined whether pseudo-haptic heaviness determined by the C/D ratio was influenced by the range of C/D ratios, and also, by the position of the C/D ratio within a given range. Participants were asked to drag and lift a square on the screen up to a target line and then rate its heaviness; the range of C/D ratios was controlled as a between-participants factor. We observed a phenomenon whereby both the range and position of the C/D ratio influenced the rated heaviness. This phenomenon was clearly established over 8 experimental trials. We conclude that both the C/D ratio range and the position of the C/D ratio within a given range are determinants for pseudo-haptic heaviness.

Force illusion induced by visual illusion: Illusory curve in cursor path is interpreted as unintended force.

Discrepancies between expected and actual visual outcomes of motor action can produce an illusory sensation of unintended force. In the present study, we addressed whether the force illusion could be induced even when the discrepancy was brought about by the illusory appearance of the actual outcome. Specifically, the apparent path of a cursor controlled by the participants was modulated by the direction of noise motion presented inside the cursor. We showed that a greater noise motion inside the cursor caused a greater apparent curve of the cursor path and, also, higher rating scores for an unintended force. We also found that the unintended force was influenced strongly by the visibility of the cursor, suggesting that the apparent curve of the cursor path was a more important factor in generating the unintended force than the noise motion itself inside the cursor. Our results suggest that the illusory force, which is mediated by cross-modal mechanisms susceptible to visual illusion, can be exploited in extended reality systems as a novel technique for giving users a sensation of force.

Underestimation in temporal numerosity judgments computationally explained by population coding model.

The ability to judge numerosity is essential to an animal's survival. Nevertheless, the number of signals presented in a sequence is often underestimated. We attempted to elucidate the mechanism for the underestimation by means of computational modeling based on population coding. In the model, the population of neurons which were selective to the logarithmic number of signals responded to sequential signals and the population activity was integrated by a temporal window. The total number of signals was decoded by a weighted average of the integrated activity. The model predicted well the general trends in the human data while the prediction was not fully sufficient for the novel aging effect wherein underestimation was significantly greater for the elderly than for the young in specific stimulus conditions. Barring the aging effect, we can conclude that humans judge the number of signals in sequence by temporally integrating the neural representations of numerosity.

The Relationship Between Illusory Heaviness Sensation and the Motion Speed of Visual Feedback in Gesture-Based Touchless Inputs.

Interaction systems with gesture-based touchless inputs are becoming more common. Nevertheless, perceptual properties of the visual feedback used in the system have not been well documented. We investigated whether the speed of motion shown in visual feedback used in gesture-based touchless inputs could be a cue for the heaviness sensation of an object even when other incidental cues, such as changes in object size and spatial consistencies in direction between gestures and feedback, were eliminated from the stimuli. Participants were asked to make a gesture to grasp and raise/lower disks shown on a horizontal display. The disk's diameter changed in accordance with the vertical position of the participant's hand. The results showed that the rate of change in diameter determined the heaviness sensation. When the disks were replaced with concentric gratings having sinusoidal radial intensity and thus the cue of size change was eliminated from the stimuli, the heaviness sensation was dependent on the speed of phase shift (that is, motion) in the grating. It was also found that spatial consistency between the direction of gestures and phase shift was not a critical condition for the heaviness sensation. Finally, the speed of motion served as a critical determinant of the heaviness sensation even when another visual feature (i.e., frame rate) was modulated in a single session, which indicates that the effect of the speed of motion on the heaviness sensation was unlikely due to demanded characteristics. The results indicate that the heaviness sensation for visual feedback of gesture-based touchless inputs is based purely on the speed of the visual feedback motion.

Delay and Speed of Visual Feedback of a Keystroke Cause Illusory Heaviness and Stiffness.

Imposing a delay between an action (e.g., a limb movement) and its related visual feedback (e.g., a cursor movement on the display) induces a peculiar sensation of heaviness or stiffness. Earlier studies have examined this delay-induced heaviness or stiffness sensation in relation to the non-arbitrary causal relationship between an action and its effect. Here, "non-arbitrary causal relationship" means that an action produces a specific and deterministic pattern of visual feedback; for example, a leftward limb movement consistently and deterministically causes a leftward visual motion. In modern graphical user interfaces, on the other hand, users often control visual information by pressing keys, wherein the relationship between the keystroke and the change in visual information is arbitrary. The present study examined whether the sensation of heaviness, stiffness and bumpiness could be caused when participants' keystroke produced a delayed arbitrary visual feedback. Participants were asked to press and hold down an assigned key to cause temporal luminance changes in a square centered on the display, an arbitrary visual feedback of their keystroke. Not only the onset delay of the temporal luminance change from the participant's keystroke but also the speed of the temporal luminance change were examined as a visual cue to heaviness, stiffness, or bumpiness. In Experiment 1, the participants' task was to give a rating for the strength of the heaviness, stiffness, or bumpiness perceived when they pressed the key. Our results showed that the heaviness and stiffness ratings increased as the delay increased and decreased as the speed increased. To check whether the manipulation of the delay and speed of the visual feedback caused changes in the subjective evaluation of sensorimotor incongruence, in Experiment 2, we asked the participants to give a rating for the sense of agency. The rating scores decreased as the delay increased and increased as the speed increased. The delay and speed influenced the rating scores for the sense of agency in the opposite direction to those for heaviness/stiffness. We discuss that the brain determines the heaviness and stiffness during a keystroke based on internalized statistics relating to the delay and speed of the action feedback.

Visual estimation of the force applied by another person.

As observers, we believe that we can visually estimate the force that another person is applying to a material. However, it is unclear what kind of cues we use to do this. We focused on two types of visual change that occur when actors push an elastic material from above with their fingers: visual shaking and visual indentation depth. The first one relates to a finger/hand shaking, known as an "induced tremor", and the second one relates to material deformation due to the application of force. We found that human observers mainly used visual shaking to estimate the force being applied by another person in a video clip. Overall, the apparent applied force was perceived to be stronger when the level of visual shaking was greater. We also found that observers mainly used visual indentation depth and visual shaking to estimate the softness rating of materials. Overall, the apparent softness was perceived to be greater when the visual indentation depth was larger and the level of visual shaking was lower, which indicates that observers use visual shaking to estimate the force being applied, and that estimated force is then used for an estimation of softness.

Perceptual judgments for the softness of materials under indentation.

Humans can judge the softness of elastic materials through only visual cues. However, factors contributing to the judgment of visual softness are not yet fully understood. We conducted a psychophysical experiment to determine which factors and motion features contribute to the apparent softness of materials. Observers watched video clips in which materials were indented from the top surface to a certain depth, and reported the apparent softness of the materials. The depth and speed of indentation were systematically manipulated. As physical characteristics of materials, compliance was also controlled. It was found that higher indentation speeds resulted in larger softness rating scores and the variation with the indentation speed was successfully explained by the image motion speed. The indentation depth had a powerful effect on the softness rating scores and the variation with the indentation depth was consistently explained by motion features related to overall deformation. Higher material compliance resulted in higher softness rating scores and these variation with the material compliance can be explained also by overall deformation. We conclude that the brain makes visual judgments about the softness of materials under indentation on the basis of the motion speed and deformation magnitude.

Sense of Resistance for a Cursor Moved by User's Keystrokes.

Haptic sensation of a material can be modulated by its visual appearance. A technique that utilizes this visual-haptic interaction is called as pseudo-haptic feedback. Conventional studies have investigated pseudo-haptic feedback in situations, wherein a user manipulated a virtual object using a computer mouse, a force-feedback device, etc. The present study investigated whether and how it was possible to offer pseudo-haptic feedback to a user who manipulated a virtual object using keystrokes. Participants moved a cursor toward a destination by pressing a key. While the cursor was moving, the cursor was temporarily slowed down on a square area of the screen. The participants' task was to report, on a five-point scale, how much resistance they felt to the cursor's movement. In addition to the basic speed of the cursor, the ratio of the basic speed to the speed within the square area was varied. In Experiment 1, we found that these two factors interacted significantly with each other, but further analysis showed that the cursor speed within the square area was the most important determinant of perceived resistance. In Experiment 2, consistent with the results of the previous experiment, it was found that the cursor movement outside of the square area was not required to generate the sense of resistance. Counterintuitively, in Experiment 3, the sense of resistance was apparent even without user's keystrokes. We discuss how the sense of resistance for a cursor moved by keystrokes can be triggered visually, but interpreted by the brain as a haptic impression.

Discounting mechanism underlies extinction illusion.

Humans can perceive a coherent visual scene despite a low spatial resolution in peripheral vision. How does the visual system determine whether an object exists in the periphery? We addressed this question by focusing on the extinction illusion in which a disk becomes subjectively invisible when presented at the intersection of grids. We hypothesized that the disk would go unnoticed when the stimuli with and without the disk produced the same strength of visual signals. The visual system would miss the disk by confounding the target signals with the intersection signals that should be discounted. Computational analysis revealed that the energy ratio between the stimuli with and without the disk decreased with stimulus eccentricity and such energy ratio could successfully explain the observer's d' to detect the disk. These results indicate that the discounting mechanism relying on stimulus energy determines the awareness toward a peripheral object.

A Computational Mechanism for Seeing Dynamic Deformation.

Human observers perceptually discriminate the dynamic deformation of materials in the real world. However, the psychophysical and neural mechanisms responsible for the perception of dynamic deformation have not been fully elucidated. By using a deforming bar as the stimulus, we showed that the spatial frequency of deformation was a critical determinant of deformation perception. Simulating the response of direction-selective units (i.e., MT pattern motion cells) to stimuli, we found that the perception of dynamic deformation was well explained by assuming a higher-order mechanism monitoring the spatial pattern of direction responses. Our model with the higher-order mechanism also successfully explained the appearance of a visual illusion wherein a static bar apparently deforms against a tilted drifting grating. In particular, it was the lower spatial frequencies in this pattern that strongly contributed to the deformation perception. Finally, by manipulating the luminance of the static bar, we observed that the mechanism for the illusory deformation was more sensitive to luminance than contrast cues.

Perceptual Properties of the Poisson Effect.

When an elastic material (e.g., fabric) is horizontally stretched (or compressed), the material is compressed (or extended) vertically - so-called the Poisson effect. In the different case of the Poisson effect, when an elastic material (e.g., rubber) is vertically squashed, the material is horizontally extended. In both cases, the visual system receives image deformations involving horizontal expansion and vertical compression. How does the brain disentangle the two cases and accurately distinguish stretching from squashing events? Manipulating the relative magnitude of the deformation of a square between horizontal and vertical dimensions in the two-dimensional stimuli, we asked observers to judge the force direction in the stimuli. Specifically, the participants reported whether the square was stretched or squashed. In general, the participant's judgment was dependent on the relative deformation magnitude. We also checked the anisotropic effect of deformation direction [i.e., horizontal vs. vertical stretching (or squashing)] and found that the participant's judgment was strongly biased toward horizontal stretching. We also observed that the asymmetric deformation pattern, which indicated the specific context of force direction, was also a strong cue to the force direction judgment. We suggest that the brain judges the force direction in the Poisson effect on the basis of assumptions about the relationship between image deformation and force direction, in addition to the relative image deformation magnitudes between horizontal and vertical dimensions.

Mid-Air Action Contributes to Pseudo-Haptic Stiffness Effects.

Pseudo-haptic feedback takes advantage of a cross-modal integration between vision and haptics. Previous studies have shown that object stiffness can be rendered with pseudo-haptic feedback with external haptic inputs. This article explored whether the pseudo-haptic feedback was feasible with a mid-air action wherein no external haptic input was given. On each trial of the experiments, participants conduced a mid-air action to laterally move their hands as if they horizontally stretched an object in the display. In synchronized with the hands' motion, the object horizontally deformed. The magnitude of the object deformation varied with the horizontal distance between participants' hands (i.e., a hand distance). The ratio of deformation magnitudes to the hand distance (i.e., a deformation-to-distance ratio) was controlled; With a larger ratio, a smaller hand distance produced the maximum level of object deformation. The Poisson's ratio was also controlled; a higher Poisson's ratio produced a larger magnitude of vertical deformation. The participants were asked to report the stiffness of the objects with a five-point rating scale. Consequently, the stiffness rating decreased with the deformation-distance ratio and with the Poisson's ratio. The results indicate that pseudo-haptic stiffness can be rendered with mid-air action by manipulating the deformation-distance ratio and Poisson's ratio.

Visual assessment of causality in the Poisson effect.

When a material is stretched along a spatial axis, it is causally compressed along the orthogonal axis, as quantified in the Poisson effect. The present study examined how human observers assess this causality. Stimuli were video clips of a white rectangular region that was horizontally stretched while it was vertically compressed, with spatially sinusoidal modulation of the magnitude of vertical compressions. It was found that the Poisson's ratio-a well-defined index of the Poisson effect-was not an explanatory factor for the degree of reported causality. Instead, reported causality was explained by image features related to deformation magnitudes. Comparing a material's shape before and after deformation was not always required for the causality assessment. This suggests that human observers determine causality in the Poisson effect by using heuristics based on image features not necessarily related to the physical properties of the material.

Perceptual Transparency From Cast Shadow.

This study examined how a shadow contributes to the perception of a transparent surface. As stimuli, we used computer graphics images in which a transparent surface with a color-mosaic pattern casts a shadow onto a background surface. We manipulated two parameters: (a) the spatial heterogeneity of the transmittance of the transparent surface and (b) the size of the light source shining on the transparent surface and its background. The latter parameter determined the blurriness of shadows. Observers judged whether the stimulus image contained a transparent surface or not. We found that the proportion of reports identifying a transparent surface was dependent on both parameters we tested. Specifically, a high spatial heterogeneity of transmittance decreased the proportion of reports of a transparent surface; this was possibly because globally defined X-junctions, which were one of the cues to perceptual transparency, perceptually broke down. On the other hand, blurred shadows were effective even when the global X-junctions were not effective. Locally defined X-junctions only moderately contributed to perceptual transparency. The results indicate that in addition to global and local X-junctions, blurred shadows are image features that elicit the perception of transparency from a cast shadow. A large individual difference as to which information each participant used as a cue to perceptual transparency was also discussed.

Perceptually Based Adaptive Motion Retargeting to Animate Real Objects by Light Projection.

A recently developed light projection technique can add dynamic impressions to static real objects without changing their original visual attributes such as surface colors and textures. It produces illusory motion impressions in the projection target by projecting gray-scale motion-inducer patterns that selectively drive the motion detectors in the human visual system. Since a compelling illusory motion can be produced by an inducer pattern weaker than necessary to perfectly reproduce the shift of the original pattern on an object's surface, the technique works well under bright environmental light conditions. However, determining the best deformation sizes is often difficult: When users try to add a large deformation, the deviation in the projected patterns from the original surface pattern on the target object becomes apparent. Therefore, to obtain satisfactory results, they have to spend much time and effort to manually adjust the shift sizes. Here, to overcome this limitation, we propose an optimization framework that adaptively retargets the displacement vectors based on a perceptual model. The perceptual model predicts the subjective inconsistency between a projected pattern and an original one by simulating responses in the human visual system. The displacement vectors are adaptively optimized so that the projection effect is maximized within the tolerable range predicted by the model. We extensively evaluated the perceptual model and optimization method through a psychophysical experiment as well as user studies.

Deformation-induced transparency resolves color scission.

Dynamic image deformation produces the perception of a transparent material that appears to deform the background image by light refraction. Since past studies on this phenomenon have mainly used subjective judgment about the presence of a transparent layer, it remains unsolved whether this is a real perceptual transparency effect in the sense that it forms surface representations, as do conventional transparency effects. Visual computation for color and luminance transparency, induced mainly by surface-contour information, can be decomposed into two components: surface formation to determine foreground and background layers, and scission to assign color and luminance to each layer. Here we show that deformation-induced perceptual transparency aids surface formation by color transparency and consequently resolves color scission. We asked observers to report the color of the front layer in a spatial region with a neutral physical color. The layer color could be seen as either reddish or greenish depending on the spatial context producing the color transparency, which was, however, ambiguous about the order of layers. We found that adding to the display a deformation-induced transparency that could specify the front layer significantly biased color scission in the predicted way if and only if the deformation-induced transparency was spatially coincident with the interpretation of color transparency. The results indicate that deformation-induced transparency is indeed a novel type of perceptual transparency that plays a role in surface formation in cooperation with color transparency.

Motion Perception: From Detection to Interpretation.

Visual motion processing can be conceptually divided into two levels. In the lower level, local motion signals are detected by spatiotemporal-frequency-selective sensors and then integrated into a motion vector flow. Although the model based on V1-MT physiology provides a good computational framework for this level of processing, it needs to be updated to fully explain psychophysical findings about motion perception, including complex motion signal interactions in the spatiotemporal-frequency and space domains. In the higher level, the velocity map is interpreted. Although there are many motion interpretation processes, we highlight the recent progress in research on the perception of material (e.g., specular reflection, liquid viscosity) and on animacy perception. We then consider possible linking mechanisms of the two levels and propose intrinsic flow decomposition as the key problem. To provide insights into computational mechanisms of motion perception, in addition to psychophysics and neurosciences, we review machine vision studies seeking to solve similar problems.